Explore data available through an OGC API, and how to filter data temporally, spatially, and by property.
Author
Tempest McCabe, Julia Signell
Published
May 23, 2023
Run this notebook
You can launch this notbook using mybinder, by clicking the button below.
Approach
Use OWSLib to determine what data is available and inspect the metadata
Use OWSLib to filter and read the data
Use geopandas and folium to analyze and plot the data
Note that the default examples environment is missing one requirement: oswlib. We can pip install that before we move on.
!pip install OWSLib==0.28.1
Requirement already satisfied: OWSLib==0.28.1 in /opt/conda/lib/python3.7/site-packages (0.28.1)
Requirement already satisfied: pyyaml in /opt/conda/lib/python3.7/site-packages (from OWSLib==0.28.1) (5.4.1)
Requirement already satisfied: lxml in /opt/conda/lib/python3.7/site-packages (from OWSLib==0.28.1) (4.9.2)
Requirement already satisfied: pytz in /opt/conda/lib/python3.7/site-packages (from OWSLib==0.28.1) (2021.1)
Requirement already satisfied: python-dateutil>=1.5 in /opt/conda/lib/python3.7/site-packages (from OWSLib==0.28.1) (2.8.2)
Requirement already satisfied: requests>=1.0 in /opt/conda/lib/python3.7/site-packages (from OWSLib==0.28.1) (2.24.0)
Requirement already satisfied: six>=1.5 in /opt/conda/lib/python3.7/site-packages (from python-dateutil>=1.5->OWSLib==0.28.1) (1.15.0)
Requirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.7/site-packages (from requests>=1.0->OWSLib==0.28.1) (2022.9.24)
Requirement already satisfied: chardet<4,>=3.0.2 in /opt/conda/lib/python3.7/site-packages (from requests>=1.0->OWSLib==0.28.1) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests>=1.0->OWSLib==0.28.1) (2.10)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /opt/conda/lib/python3.7/site-packages (from requests>=1.0->OWSLib==0.28.1) (1.25.11)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 22.0.3; however, version 23.1.2 is available.
You should consider upgrading via the '/opt/conda/bin/python3.7 -m pip install --upgrade pip' command.
from owslib.ogcapi.features import Featuresimport geopandas as gpdimport datetime as dtfrom datetime import datetime, timedelta
About the Data
The fire data shown is generated by the FEDs algorithm. The FEDs algorithm tracks fire movement and severity by ingesting observations from the VIIRS thermal sensors on the Suomi NPP and NOAA-20 satellites. This algorithm uses raw VIIRS observations to generate a polygon of the fire, locations of the active fire line, and estimates of fire mean Fire Radiative Power (FRP). The VIIRS sensors overpass at ~1:30 AM and PM local time, and provide estimates of fire evolution ~ every 12 hours. The data produced by this algorithm describe where fires are in space and how fires evolve through time. This CONUS-wide implementation of the FEDs algorithm is based on Chen et al 2020’s algorithm for California.
The data produced by this algorithm is considered experimental.
Look at the data that is availible through the OGC API
The datasets that are distributed throught the OGC API are organized into collections. We can display the collections with the command:
We will focus on the public.eis_fire_snapshot_fireline_nrt collection, the public.eis_fire_snapshot_perimeter_nrt collection, and the public.eis_fire_lf_perimeter_archive collection here.
Inspect the metatdata for public.eis_fire_snapshot_perimeter_nrt collection
We can access information that describes the public.eis_fire_snapshot_perimeter_nrt table.
It is always a good idea to do any data filtering as early as possible. In this example we know that we want the data for particular spatial and temporal extents. We can apply those and other filters using the OWSLib package.
In the below example we are:
choosing the public.eis_fire_snapshot_perimeter_nrt collection
subsetting it by space using the bbox parameter
subsetting it by time using the datetime parameter
filtering for fires over 5km^2 and over 2 days long using the filter parameter. The filter parameter lets us filter by the columns in ‘public.eis_fire_snapshot_perimeter_nrt’ using SQL-style queries.
NOTE: The limit parameter desginates the maximum number of objects the query will return. The default limit is 10, so if we want to all of the fire perimeters within certain conditions, we need to make sure that the limit is large.
## Get the most recent fire perimeters, and 7 days before most recent fire perimetermost_recent_time =max(*perm["extent"]["temporal"]["interval"])now = dt.datetime.strptime(most_recent_time, "%Y-%m-%dT%H:%M:%S+00:00")last_week = now - dt.timedelta(weeks=1)last_week = dt.datetime.strftime(last_week, "%Y-%m-%dT%H:%M:%S+00:00")print("Most Recent Time =", most_recent_time)print("Last week =", last_week)
Most Recent Time = 2023-05-23T00:00:00+00:00
Last week = 2023-05-16T00:00:00+00:00
perm_results = w.collection_items("public.eis_fire_snapshot_perimeter_nrt", # name of the dataset we want bbox=["-106.8", "24.5", "-72.9", "37.3"], # coodrinates of bounding box, datetime=[last_week +"/"+ most_recent_time], # date range limit=1000, # max number of items returnedfilter="farea>5 AND duration>2", # additional filters based on queryable fields)
The result is a dictionary containing all of the data and some summary fields. We can look at the keys to see what all is in there.
For instance you can check the total number of matched items and make sure that it is equal to the number of returned items. This is how you know that the limit you defined above is high enough.
In addition to all the summary fields, the perm_results dict contains all the data. We can pass the data into geopandas to make it easier to interact with.
Then we’ll use those fields to get most recent fire perimeters and fire lines.
perm_results = w.collection_items("public.eis_fire_snapshot_perimeter_nrt", datetime=most_recent_time, limit=1000,)perimeters = gpd.GeoDataFrame.from_features(perm_results["features"])## Get the most recent fire linesperimeter_ids = perimeters.fireid.unique()perimeter_ids =",".join(map(str, perimeter_ids))fline_results = w.collection_items("public.eis_fire_snapshot_fireline_nrt", limit=1000,filter="fireid IN ("+ perimeter_ids+")", # only the fires from the fire perimeter query above)fline = gpd.GeoDataFrame.from_features(fline_results["features"])
Make this Notebook Trusted to load map: File -> Trust Notebook
Visualize the Growth of the Camp Fire
We may be interested in understanding how a fire evolved through time. To do this, we can work with the “Large fire” or “lf” perimeter collections. The public.eis_fire_lf_perimeter_nrt colelction has the full spread history of fires from this year. public.eis_fire_lf_perimeter_archive has the full spread history of fires from 2018-2021 that were in the Western United States. The Camp Fire was in 2018, so we will work with the public.eis_fire_lf_perimeter_archive collection.
We can start by querying with information specific to the Camp Fire, like it’s genreal region (Northern California), and when it was active (November 2018).
perimeters_archive_results = w.collection_items("public.eis_fire_lf_perimeter_archive", bbox=["-124.52", "39.2", "-120", "42"], # North California bounding box, datetime=["2018-11-01T00:00:00+00:00/2018-11-30T12:00:00+00:00"], limit=3000, filter="duration>2", # additional filters based on queryable fields)perimeters_archive_resultsperimeters = gpd.GeoDataFrame.from_features(perimeters_archive_results["features"])perimeters = perimeters.sort_values(by ="t", ascending =False)perimeters = perimeters.set_crs("epsg:4326")m = perimeters.explore(style_kwds = {'fillOpacity':0})m
Make this Notebook Trusted to load map: File -> Trust Notebook
Download Data
Downloading pre-filtered data may be useful for working locally, or for working with the data in GIS software.
We can download the dataframe we made by writing it out into a shapefile or into a GeoJSON file.
The API hosts 9 different collections. There are four different types of data, and three different time-scales availible for querying through the API. “*snapshot*” collections are useful for visualizing the most recent data. It contains the most recent fires perimeters, active firelines, or VIIRS observations within the last 20 days. “*lf*” collections (short for Large Fire), show every fire perimeter, active fire line, or VIIRS observations for fires over 5 km^2. Collections that end in *archive are for year 2018 - 2021 across the Western United States. Collections with the *nrt ending are for CONUS from this most recent year. FireIDs are consistent only between layers with the same timescale (snapshot, lf_*nrt, and lf_archive*).
public.eis_fire_snapshot_perimeter_nrt
Perimeter of cumulative fire-area. Most recent perimeter from the last 20 days.
public.eis_fire_lf_perimeter_nrt
Perimeter of cumulative fire-area, from fires over 5 km^2. Every fire perimeter from current year to date.
public.eis_fire_lf_perimeter_archive
Perimeter of cumulative fire-area, from fires over 5 km^2 in the Western United States. Every fire perimeter from 2018-2021.
Column
Description
Unit
meanfrp
Mean fire radiative power. The weighted sum of the fire radiative power detected at each new pixel, divided by the number of pixels. If no new pixels are detected, meanfrp is set to zero.
MW/(pixel area)
t
Time of VIIRS detection, corrected to noon and midnight.
Datetime. yyyy-mm-ddThh:mm:ss. Local time.
fireid
Fire ID. Unique for each fire. Matches fireid.
Numeric ID
pixden
Number of pixels divided by area of perimeter.
pixels/Km^2
duration
Number of days since first observation of fire. Fires with a single observation have a duration of zero.
Days
flinelen
Length of active fire line, based on new pixels. If no new pixels are detected, flinelen is set to zero.
Km
fperim
Length of fire perimeter.
Km
farea
Area within fire perimeter.
Km^2
n_newpixels
Number of pixels newly detected since last overpass.
pixels
n_pixels
Number of pixel-detections in history of fire.
pixels
isactive
Have new fire pixels been detected in the last 5 days?
Boolean
ogc_fid
The ID used by the OGC API to sort perimeters.
Numeric ID
geometry
The shape of the perimeter.
Geometry
public.eis_fire_snapshot_fireline_nrt
Active fire line as estimated by new VIIRS detections. Most fire line from the last 20 days.
public.eis_fire_lf_fireline_nrt
Active fire line as estimated by new VIIRS detections, from fires over 5 km^2. Every fire line from current year to date.
public.eis_fire_lf_fireline_nrt
Active fire line as estimated by new VIIRS detections, from fires over 5 km^2 in the Western United States. Every fire line from 2018-2021.
Column
Description
Unit
fireid
ID of fire pixel associated with.
Numeric ID
t
Time of VIIRS detection, corrected to noon and midnight.
Datetime. yyyy-mm-ddThh:mm:ss. Local time.
mergeid
ID used to connect pixels to perimeters. Matches fireid
Numeric ID
ogc_fid
The ID used by the OGC API to sort pixels.
Numeric ID
public.eis_fire_snapshot_newfirepix_nrt
New pixel detections that inform the most recent time-step’s perimeter and fireline calculation from the last 20 days.
public.eis_fire_lf_newfirepix_nrt
New pixel detections that inform a given time-step’s perimeter and fireline calculation. Availible from start of current year to date.
public.eis_fire_lf_newfirepix_archive
New pixel detections that inform a given time-step’s perimeter and fireline calculation. Availible for Western United States from 2018-2021.
Column
Description
Unit
fireid
ID of fire pixel associated with.
Numeric ID
t
Time of VIIRS detection, corrected to noon and midnight.
Datetime. yyyy-mm-ddThh:mm:ss. Local time.
mergeid
ID used to connect pixels to perimeters. Matches fireid
